Image processing apparatus and method

ABSTRACT

An image processing apparatus is disclosed. A calculation unit may determine a disocclusion region of a first frame of a video. A processing unit may generate color information of the disocclusion region of the first frame using color information associated with a second frame of the video.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/382,747 filed Mar. 23, 2009, which claims the benefit of KoreanPatent Application No. 10-2008-0099799, filed on Oct. 10, 2008, in theKorean Intellectual Property Office, the disclosure of which isincorporated herein by reference.

BACKGROUND

1. Field

Example embodiments relate to a virtual three-dimensional (3D) display,and more particularly, to an apparatus and method which stores colorinformation of a disocclusion region based on a viewpoint in amulti-view display.

2. Description of the Related Art

A virtual three-dimensional (3D) display provides different images toleft and right eyes of a human, and thereby may provide a 3D effect.Here, the different images may indicate images from differentviewpoints. In a stereoscopic method, a virtual 3D display may providetwo different images, that is, one image for a left eye and anotherimage for a right eye. In a 3D multi-view, a plurality of imagesdifferent from each other may be provided depending on a viewing anglewith a display.

Also, inputted color information and depth information of a particularframe may be used to render a 3D multi-view image. In this instance, adisocclusion phenomenon is to be overcome. The depth information mayinclude information about an object with a relatively significantdisparity and information about a background with a relativelyinsignificant disparity. When rendering is performed at a plurality ofviewpoints for the 3D multi-view, a disocclusion region having no colorinformation may be generated in a region having a large viewing angle.

A technology to overcome a disocclusion region is required due to thedevelopment of a multi-view display.

SUMMARY

Example embodiments may provide an image processing apparatus and methodwhich effectively predicts a disocclusion region and generates colorinformation.

Example embodiments may also provide an image processing apparatus andmethod which efficiently obtains color information of a disocclusionregion from color information of at least one frame of a video.

According to example embodiments, there may be provided an imageprocessing apparatus, including a calculation unit to determine adisocclusion region of a first frame of a video, and a processing unitto generate color information of the disocclusion region of the firstframe using color information associated with a second frame of thevideo.

The calculation unit may determine the disocclusion region of the firstframe based on a difference between a disparity of a background area ofthe first frame and a disparity of an object area of the first frame. Inthis instance, the disparities may be calculated using depth informationassociated with the first frame.

The calculation unit may determine a boundary between the object areaand the background area of the first frame, and determine a band with afirst width around the boundary as the disocclusion region of the firstframe. In this instance, the first width may be in proportion to thedifference between the disparity of the background area of the firstframe and the disparity of the object area of the first frame.

At least one of the disparity of the background area of the first frameand the disparity of the object area of the first frame may becalculated between a viewpoint having a maximum viewing angle and aviewpoint having a minimum viewing angle from among a plurality ofviewpoints associated with a multi-view image to be rendered based onthe first frame.

The processing unit may determine a first block around the disocclusionregion of the first frame, determine a first characteristic value of anarea corresponding to the first block within at least one frame of thevideo, and determine the second frame based on the first characteristicvalue.

The processing unit may determine, as candidate frames of the secondframe, a frame where the first characteristic value is less than a firstthreshold value from among the at least one frame of the video, generatecolor information of the disocclusion region around the first block withrespect to the candidate frames of the second frame, and determine aframe which generates the greatest amount of color information of thedisocclusion region around the first block, as the second frame. Theprocessing unit may compare the first characteristic value of each ofthe candidate frames of the second frame to determine the second frame.

The processing unit may obtain color information associated with pixelshaving a depth value difference with pixels of the background area ofthe first frame from among pixels around the area corresponding to thefirst block within the candidate frames of the second frame, determinethe color information of the disocclusion region around the first block,and determine the frame which generates the greatest amount of colorinformation of the disocclusion region around the first block, as thesecond frame. In this instance, the depth value difference may be lessthan a second threshold value.

The processing unit may copy color information of an area around thearea corresponding to the first block within the second frame togenerate the color information of the disocclusion region of the firstframe. In this instance, the processing unit may copy the colorinformation of the area around the area corresponding to the first blockwithin the second frame, change a location of the first block in thefirst frame, and copy color information of an area around an areacorresponding to the changed first block within the second frame togenerate the color information of the disocclusion region of the firstframe.

The image processing apparatus may further include a rendering unit torender a multi-view image based on color information associated with thefirst frame, depth information associated with the first frame, thecolor information of the disocclusion region of the first frame.

According to example embodiments, there may be provided an imageprocessing apparatus, including a calculation unit to determine adisocclusion region of a first frame of a video, and a processing unitto generate color information of the disocclusion region of the firstframe using any one of color information associated with a second frameof the video and color information associated with the first frame ofthe video.

According to example embodiments, there may be provided an imageprocessing method, including, determining a disocclusion region of afirst frame of a video, and generating color information of thedisocclusion region of the first frame using color informationassociated with a second frame of the video.

The generating may determine a first block around the disocclusionregion of the first frame, determine a first characteristic value of anarea corresponding to the first block within at least one frame of thevideo, and determine the second frame based on the first characteristicvalue.

Additional aspects, features, and/or advantages of example embodimentswill be set forth in part in the description which follows and, in part,will be apparent from the description, or may be learned by practice ofthe disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects, and advantages of example embodiments willbecome apparent and more readily appreciated from the followingdescription, taken in conjunction with the accompanying drawings ofwhich:

FIG. 1 illustrates a conceptual diagram of an image processing apparatusaccording to example embodiments;

FIG. 2 illustrates color information and depth information of a firstframe according to example embodiments;

FIG. 3 illustrates disocclusion regions to be restored by an imageprocessing apparatus according to example embodiments;

FIG. 4 illustrates an image processing apparatus according to exampleembodiments;

FIG. 5 illustrates a method of obtaining a disparity of a first frame,for example, the first frame of FIG. 1;

FIG. 6 illustrates disocclusion regions of a first frame, for example,the first frame of FIG. 2;

FIG. 7 illustrates a first block to generate color information of adisocclusion region, for example, the disocclusion region of FIG. 6;

FIG. 8 illustrates a method of generating color information of adisocclusion region of a first frame in a processing unit, for example,the processing unit of FIG. 4;

FIG. 9 illustrates color information of disocclusion regions of a firstframe according to example embodiments; and

FIG. 10 illustrates multi-view images provided by an image processingapparatus according to example embodiments.

DETAILED DESCRIPTION

Reference will now be made in detail to example embodiments, examples ofwhich are illustrated in the accompanying drawings, wherein likereference numerals refer to the like elements throughout. Exampleembodiments are described below to explain the present disclosure byreferring to the figures.

FIG. 1 illustrates an image processing apparatus 100 according toexample embodiments.

A frame 110 is an i^(th) frame of a video for three-dimensional (3D)rendering. Here, i may be a natural number, and a plurality of framesmay be provided through the video. The image processing apparatus 100may generate color information of a disocclusion region of the frame110. Hereinafter, the frame 110 is referred to as a first frame 110.

Color information 111 may indicate a color value such as a Red, Greenand Blue (RGB) value, associated with the first frame 110. Depthinformation 112 may include a depth value of pixels associated with thefirst frame 110. A rendering unit 120 may match the depth information112 with the color information 111.

A multi-view screen (or a display panel) 130 may provide differentimages to a plurality of viewpoints by refracting multi-view imagesprovided from the rendering unit 120. The multi-view screen 130 mayinclude a lenticular lens.

An image 141 may be observed at a first viewpoint. The first viewpointmay have a maximum viewing angle from among the plurality of viewpointsin a leftmost side.

An image 142 may be observed at a viewpoint having a minimum viewingangle with the multi-view screen 130 from among the plurality ofviewpoints. Accordingly, the image 142 may be in an orthogonal directionto the multi-view screen 130.

An image 143 may be observed at a k^(th) viewpoint. The k^(th) viewpointmay have a maximum viewing angle from among the plurality of viewpointsin a rightmost side. k may be associated with a specification of themulti-view screen 130, and be a total number of the plurality ofviewpoints. For example, when the multi-view screen 130 has nine-viewcharacteristics, k may be nine.

However, a first disocclusion region which is rendered as black may beshown in the image 141. The first disocclusion region may be generatedsince color information of a portion of a background does not exist in aleft side of a die which is an object.

A second disocclusion region which is rendered as black may be shown inthe image 143. The second disocclusion region may be generated sincecolor information of a portion of the background does not exist in aright side of the object. A disocclusion region is not shown in theimage 142.

When k multi-view images are provided and two images neighboring eachother are provided to both eyes of an observer, k−1 3D images may beprovided. In this instance, the k−1 3D images may be different from eachother. Also, a left image of the two images may be provided to a lefteye and a right image of the two images may be provided to a right eye.Accordingly, the observer may recognize the object as a 3D object, andexperience an effect as though the observer looks around the object maybe obtained.

According to example embodiments, the image processing apparatus 100 maybe provided with the color information 111 and the depth information 112of the first frame 110, and color information of frames excluding thefirst frame 110, and thereby may generate color information of thedisocclusion region of the first frame 110, and provide the colorinformation to the rendering unit 120.

According to example embodiments, the image processing apparatus 100 maygenerate the color information of the disocclusion region of the firstframe 110 based on color information of a second frame from among theframes excluding the first frame 110. However, the image processingapparatus 100 may not be limited to the above-described embodiment, andmay also generate a portion of the color information of the disocclusionregion of the first frame 110 based on the color information 111 of thefirst frame 110.

The rendering unit 120 may generate k (multi-view) images using thecolor information 111 and the depth information 112 of the first frame110, and the color information of the disocclusion region of the firstframe 110.

An image 151 generated by the image processing apparatus 100 may beprovided to the first viewpoint. The image 151 may correspond to theimage 141, and have color information of the first disocclusion regionof the image 141.

An image 153 generated by the image processing apparatus 100 may beprovided to the k^(th) viewpoint. The image 153 may correspond to theimage 143, and may have color information corresponding to the seconddisocclusion region of the image 143.

The image processing apparatus 100 and method is described in greaterdetail with reference to FIG. 2 through FIG. 10.

FIG. 2 illustrates color information and depth information of a firstframe according to example embodiments.

Color information 210 may correspond to the color information 111 ofFIG. 1. A region 211 may correspond to the disocclusion region of theimage 141. The region 211 may be occluded by the object at a viewpointof FIG. 2. Also, a region 212 may correspond to the disocclusion regionof the image 143. Although the region 212 may be occluded by the objectat the viewpoint of FIG. 2, the region 212 may be shown depending on aviewpoint where rendering is performed. Accordingly, the region 212requires color information.

Depth information 220 may indicate a distance from the multi-view screen130. An object area 221 may have a great depth value, and thus theobject area 221 is bright. A background area 222 may have a small depthvalue, and thus the background area 222 is dark. The depth values areincluded in the depth information 220. Although the depth information220 is simply illustrated for convenience of description, the depthinformation 220 may generally include a number of levels (or agradient).

FIG. 3 illustrates disocclusion regions to be restored by an imageprocessing apparatus according to example embodiments.

An image 310 may correspond to the image 141 of FIG. 1. As describedabove, when comparing to other images 320, 330, 340, 350, and 360, theimage 310 may have a disocclusion region 311 with a greatest size in aleft side of the object. An image 320 may correspond to the imageprovided at a second viewpoint, wherein the second viewpoint may be oneof the multiple viewpoints illustrated in FIG. 1. The second viewpointmay have a large viewing angle in a second left side of the object. Theimage 320 may have a disocclusion region 321 smaller than thedisocclusion region 311 of the image 310. Similarly, an image 330 maycorrespond to a fourth viewpoint, wherein the fourth viewpoint may beone of the multiple viewpoints in FIG. 1, and have a disocclusion region331 smaller than the disocclusion region 321.

Also, when the total number of views k, is nine, and when a viewingangle is the smallest, an image 340 may correspond to the image 142 ofFIG. 1, and a disocclusion region is not generated in the image 340.

When a viewpoint moves to the left side of the object, an image 350 andthen another image 360 may be observed. A disocclusion region 351 existsin a right side of the object in the image 350, and a disocclusionregion 361 with a maximum size exists in the right side of the object inthe other image 360.

The image processing apparatus 100 may determine the disocclusion region311 and the disocclusion region 361, and generate color information.

FIG. 4 illustrates an image processing apparatus 400 according toexample embodiments.

A calculation unit 410 may calculate a difference between a disparity ofan object area and a disparity of a background area at a plurality ofviewpoints of a first frame, and thereby may determine a disocclusionregion of the first frame.

According to example embodiments, the calculation unit 410 may receivedepth information of the first frame. Also, the calculation unit 410 maydetermine a boundary between the object area and the background areabased on a depth value of each area and/or a disparity. The depth valuemay be included in the depth information, and the disparity may becalculated at an arbitrary viewpoint. Also, the calculation unit 410 maydetermine the disocclusion region of the first frame around theboundary, which is described in greater detail with reference to FIG. 5.

A processing unit 420 may receive information about the disocclusionregion of the first frame from the calculation unit 410, and determine afirst block around the disocclusion region. The first block may be usedto retrieve a reference frame for generating color information of adisocclusion region, for example, the disocclusion region 610 of FIG. 6.

The processing unit 420 may receive color information associated with aplurality of frames. Also, the processing unit 420 may retrieve an areacorresponding to the first block in each of the plurality of frames. Thearea may be matched with the first block due to similar color values.

According to example embodiments, the processing unit 420 may calculatea first characteristic value. The first characteristic value may be aSum of Squared Difference (SSD), and be calculated based on a colorvalue of a predetermined block. As an SSD between predetermined areas oftwo frames decreases, the predetermined areas may approach a same colorvalue.

Also, the processing unit 420 may compare the first characteristic valueof each of the plurality of frames to a first threshold value, anddetermine a frame where the first characteristic value is less than thefirst threshold value as a candidate frame of a second frame. The secondframe may be a reference frame, and a plurality of candidate frames ofthe second frame may exist.

According to example embodiments, the processing unit 420 may determine,as the second frame, a frame which may provide the greatest amount ofcolor information of a right area of an area corresponding to the firstblock from among the candidate frames of the second frame. In thisinstance, when other conditions are the same, a frame having a smallerSSD may be determined as the second frame. In this instance, the SSD maybe an SSD between predetermined areas of the first frame and the secondframe.

According to example embodiments, when a regular repetition is shown incolor information of the background area of the first frame according toa characteristic of a texture, the processing unit 420 may generatecolor information of at least a portion of the disocclusion region ofthe first frame.

According to example embodiments, the image processing apparatus 400 mayfurther include a rendering unit 430. The rendering unit 430 may beprovided with color information and depth information associated withthe first frame, and color information of the disocclusion region of thefirst frame, and thereby may provide a multi-view color image through amulti-view screen 130.

FIG. 5 illustrates a method of obtaining a disparity of a first frame,for example the first frame 110 of FIG. 1.

A disparity 521, represented in Equation 1 below as d, may indicate adifference of locations where a point 511 is projected to a screen 520when a left eye 531 and a right eye 532 look at the point 511. Since adistance, E, 533 between the left eye 531 and the right eye 532 islimited, the disparity 521 may increase, as a distance D between thescreen 520 and the line 510 containing the point 511 is greater than adistance V between the screen 520 and a line 530 containing the pointlocations of the left eye 531 and the right eye 532. That is, as anobject is far from the left eye 531 and the right eye 532, the disparity521 may decrease. Also, as the object is close to the left eye 531 andthe right eye 532, the disparity 521 may increase. The disparity 521 maybe calculated by, for example, Equation 1, shown below.

$\begin{matrix}{d = \frac{D \times E}{D + V}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

According to example embodiments, the calculation unit 410 (FIG. 4) maycalculate a difference between a disparity of a background area and adisparity of an object area at the first viewpoint and a fifthviewpoint, in selecting for example, two of the multiple viewpointsillustrated in FIG. 1, and thereby may determine a disocclusion regionin a left side of the object. In this instance, the first viewpoint mayhave a maximum viewing angle and the fifth viewpoint may be a viewpointwhen k is nine. Also, the disparity of the object area may be relativelysignificant, and the disparity of the background area may be relativelyinsignificant.

Also, the calculation unit 410 (FIG. 4) may calculate a differencebetween a disparity of the background area and a disparity of the objectarea at the fifth viewpoint and a ninth viewpoint, in selecting forexample, two of the multiple viewpoints illustrated in FIG. 1, andthereby may determine a disocclusion region in a right side of theobject. In this instance, the fifth viewpoint and the ninth viewpointmay be viewpoints when k is nine.

According to example embodiments, and referring again to FIGS. 1, 2, and4, the calculation unit 410 may receive the depth information 112 of thefirst frame 110. Also, the calculation unit 410 may determine a boundarybetween the object area 221 and the background area 222 based on a depthvalue and/or a disparity. In this instance, the depth value may beincluded in the depth information 112 and the disparity may becalculated at an arbitrary viewpoint. Also, the calculation unit 410 maydetermine disocclusion regions 211 and 212 of the first frame 110 aroundthe boundary. In this instance, a plurality of boundaries may exist.

According to example embodiments, the calculation unit 410 may extract avertical boundary from the boundaries. When the number of the extractedvertical boundaries is more than one, the calculation unit 410 mayselect a vertical boundary in a left half plane of the first frame 110.Then, the calculation unit 410 may determine a band to the right of theselected vertical boundary. The band may have a first width, and theregion including the band may be determined as the left disocclusionregion 211. Also, the calculation unit 410 may select a verticalboundary in a right half plane, and determine a band to the left of theselected vertical boundary. The band may have the first width and theregion including the band may be determined as the right disocclusionregion 212.

The first width may be calculated by, for example, Equation 2, shownbelow.

$\begin{matrix}{{{First}\mspace{14mu} {width}} = {\frac{{D({object})} \times \frac{k}{2} \times E}{{D({object})} + V} - \frac{{D({background})} \times \frac{k}{2} \times E}{{D({background})} + V}}} & {{Equation}\mspace{14mu} 2}\end{matrix}$

Here, a first term of a right side may be the disparity of the objectarea calculated by Equation 1, and a second term of the right side maybe the disparity of the background area calculated by Equation 1. InEquation 2, E may denote a distance between viewpoints neighboring toeach other from among k viewpoints. In this instance, k may denote anumber of multi-view viewpoints. The number of viewpoints k may bedivided by two to obtain a difference between a disparity between aviewpoint having a maximum viewing angle and a viewpoint having aminimum viewing angle.

According to example embodiments, although the band with the first widthmay be determined as the disocclusion region of the first frame 110around a selected boundary, the image processing apparatus and methodmay not be limited to the described example embodiments. Althoughcomplexity of computation may increase, a disocclusion region may beadaptively determined in each area.

FIG. 6 illustrates disocclusion regions of a first frame, for example,the first frame of FIG. 2.

A calculation unit 410 (FIG. 4) may determine a left disocclusion region610 and a right disocclusion region 620 in a first frame 600.

According to example embodiments, color information of the leftdisocclusion region 610 and the right disocclusion region 620 may beseparately or simultaneously generated by a processing unit 420 (FIG.4). Specifically, when n disocclusion regions exist, the processing unit420 (FIG. 4) may separately generate the color information of each ofthe left disocclusion region 610 and the right disocclusion region 620,or simultaneously generate the color information of a portion or alldisocclusion regions.

FIG. 7 illustrates a first block to generate color information of adisocclusion region, for example, the disocclusion region 610 of FIG. 6.

According to example embodiments, the processing unit 420 (FIG. 4) mayreceive information about the disocclusion region 610 of FIG. 6. Also,the processing unit 420 (FIG. 4) may determine a first block 710 aroundthe disocclusion region 610 (FIG. 6). The first block 710 may be used toretrieve a reference frame to generate the color information of thedisocclusion region 610 (FIG. 6).

The processing unit 420 (FIG. 4) may receive color informationassociated with a plurality of frames such as an i−nth frame,i-(n+1)^(th) frame, . . . , i−1^(th) frame, i^(th) frame, i+1^(th)frame, . . . , i+n^(th) frame. Also, the processing unit 420 (FIG. 4)may retrieve an area corresponding to the first block 710 in each of theplurality of frames. In this instance, the area may be matched with thefirst block 710 due to similar color values.

According to example embodiments, the processing unit 420 (FIG. 4) maycalculate a first characteristic value. The first characteristic valuemay be an SSD, and be calculated based on a color value of apredetermined block. As an SSD between predetermined areas of two framesdecreases, the areas may approach a same color value.

Also, the processing unit 420 (FIG. 4) may compare the firstcharacteristic value of each of the plurality of frames to a firstthreshold value, and determine a frame where the first characteristicvalue is less than the first threshold value as a candidate frame of asecond frame. The second frame may be a reference frame, and a pluralityof candidate frames of the second frame may exist.

According to example embodiments, the processing unit 420 (FIG. 4) maygenerate color information of the disocclusion region based on a colorvalue of a right area of an area corresponding to the first block fromamong the candidate frames of the second frame. However, the colorinformation of the disocclusion region may be generated from colorinformation of an area corresponding to a background area by referringto depth information of each of the plurality of frames. In thisinstance, the depth information of each of the plurality of frames maybe provided to the processing unit 420 (FIG. 4), and the processing unit420 may use the depth information. Also, depth values of the first frameand each of the candidate frames of the second frame may be compared,and color information of only an area where a depth value is equal to orless than a second threshold value may be used to generate the colorinformation of the disocclusion region.

Accordingly, when a location of an object may change depending on aframe, the disocclusion region may be completely shown, and thus a framehaving color information of the disocclusion region may be determined asthe second frame. That is, the processing unit 420 (FIG. 4) maydetermine, as the second frame, a frame which may provide the greatestamount of color information of a right area of the area corresponding tothe first block from among the candidate frames of the second frame. Inthis instance, when other conditions are the same, a frame having asmaller SSD may be determined as the second frame. In this instance, theSSD may be an SSD between predetermined areas of the first frame and thesecond frame.

According to example embodiments, when a regular repetition is shown incolor information of a background area of a first frame 700 according toa characteristic of a texture, the processing unit 420 (FIG. 4) maygenerate color information of a disocclusion region from the first frame700.

Although a first block 710 is illustrated as an example in FIG. 7,embodiments may not be limited to the described example embodiments.That is, it would be appreciated by those skilled in the related artthat changes may be made to these example embodiments. For example, asize and location of the first block 710 next to a disocclusion regionin the first frame 700 may be adapted depending on a frame, requiredquality, characteristics of a video, and the like.

FIG. 8 illustrates a method of generating color information of adisocclusion region of a first frame in a processing unit, for example,the processing unit 420 of FIG. 4.

The method of generating color information of a disocclusion region of afirst frame may be embodied by the processing unit 420 of FIG. 4.

In operation S810, the processing unit 420 (FIG. 4) may determine afirst block around the disocclusion region based on information aboutthe disocclusion region. In this instance, the first block may be thefirst block 710 of FIG. 7 and the disocclusion region may be thedisocclusion region 610 of FIG. 6. Also, the processing unit 420 maycalculate a first characteristic value. For example, the firstcharacteristic value may be an SSD between the first frame and a j^(th)frame. In this instance, j may be a natural number. As the SSD betweenpredetermined areas of the first frame and the j^(th) frame decreases,the first frame and the j^(th) frame may have a same color value.

In operation S820, the processing unit 420 (FIG. 4) may compare apredetermined first threshold value, Th, to the first characteristicvalue of each of a plurality of frames. When the first characteristicvalue is greater than the first threshold value, the processing unit 420(FIG. 4) may ignore the j^(th) frame, and return to operation S810 withrespect to a subsequent frame.

If the first characteristic value is less than the first thresholdvalue, then the method proceeds to operation S830, where the processingunit 420 (FIG. 4) may determine the j^(th) frame as a candidate frame ofa second frame. In this instance, a plurality of candidate frames mayexist.

In operation S840, the processing unit 420 (FIG. 4) may determinewhether the j^(th) frame is a last frame to check. When the j^(th) frameis not the last frame, the processing unit 420 (FIG. 4) returns tooperation S810, and may perform the determining in operation S810, thecomparing in operation S820, and the determining in operation S830again.

If the j^(th) frame is the last frame, then the method proceeds tooperation S850, where the processing unit 420 (FIG. 4) may select thesecond frame from the candidate frames of the second frame. In operationS860, the processing unit 420 (FIG. 4) may generate color information ofa disocclusion region of the first frame. The selecting in operationS850 and the generating in operation S860 has been described above withreference to FIG. 7.

According to example embodiments, the disocclusion region of the firstframe may be divided into a plurality of disocclusion regions and colorinformation of any one of the plurality of the disocclusion regions maybe generated in operation S810 through operation S860. When the methodis performed with respect to all the disocclusion regions, colorinformation of the entire disocclusion region may be generated.

According to example embodiments, when the color information of theentire disocclusion regions of the first frame is not generated inoperation S810 through operation S860, color information may becompletely generated through extrapolation.

FIG. 9 illustrates color information of disocclusion regions 910 and 920of a first frame 900 according to example embodiments.

The color information of the disocclusion region 910 in the first frame900 may correspond to color information of the left disocclusion region610 of FIG. 6. Also, the color information of the disocclusion region920 may be color information of the right disocclusion region 620.

FIG. 10 illustrates multi-view images provided by an image processingapparatus according to example embodiments.

Referring to FIGS. 1, 4, 9 and 10, the color information 910 and 920,generated by the processing unit 410, as well as the color information111 and the depth information 112 may be provided to the rendering unit120. In this instance, the color information 111 and the depthinformation 112 may be associated with the first frame 110 of FIG. 1.Also, as an example, when k is nine, the rendering unit 120 may provideimages 1010, 1020, 1030, 1040, 1050, and 1060 at nine viewpoints throughthe multi-view screen 130.

The images 1010, 1020, 1030, 1040, 1050, and 1060 may correspond to theimages 310, 320, 330, 340, 350, and 360 of FIG. 3, respectively. Whencomparing each of the images 1010, 1020, 1030, 1040, 1050, and 1060, toeach of the images 310, 320, 330, 340, 350, and 360, it may beascertained that the disocclusion regions 311, 321, 331, 351, and 361are improved.

The image processing method according to the above-described exampleembodiments may be recorded in computer-readable media including programinstructions to implement various operations embodied by a computer. Themedia may also include, alone or in combination with the programinstructions, data files, data structures, and the like. Examples ofcomputer-readable media include magnetic media such as hard disks,floppy disks, and magnetic tape; optical media such as CD ROM disks andDVDs; magneto-optical media such as optical disks; and hardware devicesthat are specially configured to store and perform program instructions,such as read-only memory (ROM), random access memory (RAM), flashmemory, etc. Examples of program instructions include both machine code,such as produced by a compiler, and files containing higher level codethat may be executed by the computer using an interpreter. The describedhardware devices may be configured to act as one or more softwaremodules in order to perform the operations of the above-describedexample embodiments, or vice versa.

Although a few example embodiments have been shown and described, itwould be appreciated by those skilled in the art that changes may bemade to these example embodiments without departing from the principlesand spirit of the disclosure, the scope of which is defined by theclaims and their equivalents.

What is claimed is:
 1. An image processing method, comprising:generating color information of a disocclusion region of a first frameusing color information associated with a second frame.